IMAGE CAPTIONING

English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning (D19-52)

Challenge:	Recent advances in language and vision have made incredible progress in describing images and interacting with visual content in a physical or embodied environment.
Approach:	This tutorial will provide an overview of the growing number of multimodal tasks and datasets that combine textual and visual understanding.
Outcome:	This tutorial will review the state-of-the-art approaches to selected tasks such as image captioning, visual question answering and visual dialog.

Copied to clipboard

Sahinur Rahman Laskar, Rohit Pratap Singh, Partha Pakray, Sivaji Bandyopadhyay

On the Role of Scene Graphs in Image Captioning (D19-64)

Challenge:	Multi-modal translation is an emerging task of the MT community, where visual features of image combine with textual features of parallel source-target text to translate sentences.
Approach:	They propose to use convolutional neural net-works and visual geometry to extract image features and attention-based Neural MachineTranslation (NMT) system for translation.
Outcome:	The proposed multi-modal translation system improves translation quality and improves the quality of the captions of the images.

Copied to clipboard

Dalin Wang, Daniel Beck, Trevor Cohn

Creative Natural Language Generation (2023.emnlp-tutorial)

Challenge:	Recent captioning approaches rely on ad-hoc approaches to obtain graphs for images, but they introduce noise and it is unclear the effect of parser errors on captioning accuracy.
Approach:	They investigate whether scene graphs can help image captioning . they show that a scene graph parser can boost performance almost as much as ground truth graphs .
Outcome:	The proposed parser can boost performance almost as much as ground truth graphs .

Copied to clipboard

Tuhin Chakrabarty, Vishakh Padmakumar, He He, Nanyun Peng

Multi-view and Cross-view Brain Decoding (2022.coling-1)

Challenge:	This tutorial aims to bring awareness of the important and emerging research area of open-domain creative generation.
Approach:	They will review recent studies on creative language generation at sentence level as well as longer forms of text.
Outcome:	This paper reviews recent studies on creative language generation at sentence level as well as longer forms of text.

Copied to clipboard

Subba Reddy Oota, Jashn Arora, Manish Gupta, Raju S. Bapi

“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?! (2021.acl-srw)

Challenge:	a recent study has shown that brain decoding models can decode concepts from single view . a multi-view decoder can take brain recordings for any view as input and predict the concept .
Approach:	They propose to build a multi-view decoder that can take brain recordings for any view as input and predict the concept.
Outcome:	The proposed systems can decode concepts from brain recordings from any view . the proposed systems have 0.68 pairwise accuracy across view pairs and 0.8 average pairwise precision across tasks.

Copied to clipboard

Alberto Testoni, Raffaella Bernardi

simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions (D18-1)

Challenge:	a problem with natural language generation systems is the generation of tokens that are unrelated to the source input.
Approach:	They propose two new models to play the GuessWhat?! referential game . they propose to adapt the best visual processing models available to mitigate this issue .
Outcome:	The proposed models generate few hallucinations compared to other models available in the literature.

Copied to clipboard

Fenglin Liu, Xuancheng Ren, Yuanxin Liu, Houfeng Wang, Xu Sun

An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024.findings-eacl)

Challenge:	Existing approaches to image captioning combine visual and semantic attention to generate a detailed and comprehensive caption.
Approach:	They propose a stepwise image-topic merging network that integrates visual and semantic attentions to generate a detailed caption.
Outcome:	The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performance.

Copied to clipboard

Saba Ahmadi, Aishwarya Agrawal

Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage (2020.acl-main)

Challenge:	Recent studies have proposed reference-free evaluations of image captions . however, these approaches are restrictive and favor captions with similar vocabulary but different meanings.
Approach:	They propose to use reference-free metrics to evaluate image captions . they propose to combine lexical overlap and semantics to identify fine-grained errors .
Outcome:	The proposed metrics struggle to identify fine-grained errors, the authors show . CLIPScore, UMIC, and PAC-S are sensitive to variations in image-relevant objects mentioned in the caption .

Copied to clipboard

Ashish V. Thapliyal, Radu Soricut

JEEM: Vision-Language Understanding in Four Arabic Dialects (2026.findings-eacl)

Challenge:	Existing datasets have extensive labeled data for En-glish, but labeles are extremely scarce in other languages.
Approach:	They propose a method that leverages existing annotations with machine translation capabilities to create cross-modal language generation systems at web-scale.
Outcome:	The proposed model outperforms other candidates in evaluations performed over 5 target languages.

Copied to clipboard

Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, Yova Kementchedjhieva

LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences (2025.acl-industry)

Challenge:	Existing evaluation datasets feature Western-centric images and English text, while their non-English counterparts are often derived from the latter.
Approach:	They propose to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco.
Outcome:	The proposed model underperforms in visual understanding and dialect-specific generation across four Arabic-speaking countries.

Copied to clipboard

Yusuke Hirota, Boyi Li, Ryo Hachiuma, Yueh-Hua Wu, Boris Ivanovic, Marco Pavone, Yejin Choi, Yu-Chiang Frank Wang, Yuta Nakashima, Chao-Han Huck Yang

O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning (2021.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework .
Approach:	They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations.
Outcome:	The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences.

Copied to clipboard

Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, Xu Sun

Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)

Challenge:	Existing methods for video captioning consider a sequence of frames and biases towards focused objects.
Approach:	They propose an Object-Oriented Non-Autoregressive approach to video captioning . it performs three steps: 1) identify the focused objects and predict their locations . 2) generate related attribute words and relation words of these focused objects to form a draft caption .
Outcome:	The proposed method achieves competitive results with the state-of-the-art methods but with higher diversity and faster inference speed.

Copied to clipboard

Ting Wang, Weidong Chen, Yuanhe Tian, Yan Song, Zhendong Mao

Enhancing Descriptive Image Captioning with Natural Language Inference (2021.acl-short)

Challenge:	Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship .
Approach:	They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals .
Outcome:	The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts.

Copied to clipboard

Zhan Shi, Hui Liu, Xiaodan Zhu

Transferring General Multimodal Pretrained Models to Text Recognition (2023.findings-acl)

Challenge:	Existing captioning models tend to generate generic captions, but generating descriptive captions is important.
Approach:	They propose a novel approach to encourage captioning models to produce more detailed captions using natural language inference.
Outcome:	The proposed method outperforms baseline models on MSCOCO metrics on descriptiveness and descriptiveness.

Copied to clipboard

Junyang Lin, Xuancheng Ren, Yichang Zhang, Gao Liu, Peng Wang, An Yang, Chang Zhou

Examining Gender and Racial Bias in Large Vision–Language Models Using a Novel Dataset of Parallel Images (2024.eacl-long)

Challenge:	Existing methods for text recognition rely on large-scale pretraining on human-annotated or synthetic data.
Approach:	They propose a method to transfer multimodal pretrained models to text recognition using image captioning.
Outcome:	The proposed method outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark.

Copied to clipboard

Kathleen Fraser, Svetlana Kiritchenko

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021.acl-long)

Challenge:	a new wave of large vision–language models (LVLMs) incorporate images as input in addition to text . a recent study examined potential gender and racial biases in such systems based on the perceived characteristics of the people in the input images.
Approach:	They examine potential gender and racial biases in large vision–language models . they query a dataset of AI-generated images of people to see whether they differ .
Outcome:	The proposed dataset shows that the images differ in gender and race according to the perceived characteristics of the person depicted.

Copied to clipboard

Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang

VizSeq: a visual analysis toolkit for text generation tasks (D19-3)

Challenge:	Existing vision-language pre-training methods use a two-step training procedure to learn visual features from image-text pairs.
Approach:	They propose a vision-language pre-trained model for V+L understanding and generation using a unified Transformer framework.
Outcome:	The proposed model can learn visual representation and semantic alignments between image and text on visual-text pairs and on visual processing tasks.

Copied to clipboard

Changhan Wang, Anirudh Jain, Danlu Chen, Jiatao Gu

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models (2024.findings-emnlp)

Challenge:	Several softwares for text evaluation are available that do not provide detailed examples.
Approach:	They propose a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks.
Outcome:	The proposed toolkit covers most common n-gram metrics and latest embedding-based metrics such as BERTScore.

Copied to clipboard

Donghoon Kim, Gusang Lee, Kyuhong Shim, Byonghyo Shim

Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers (2024.naacl-short)

Challenge:	Large multi-modal models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across multi-dimensional applications.
Approach:	They propose a parameter-efficient fine-tuning strategy that combines both . they find that parameter tuning methods distort the feature representation space .
Outcome:	The proposed strategy preserves representation space while limiting performance on downstream tasks.

Copied to clipboard

Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, Arash Eshghi

The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)

Challenge:	Recent approaches for developing vision and language models leverage existing vision and a language expert and try to learn a mapping between them.
Approach:	They propose to use a resampler module to create a ‘visual prompt’ which is then fed to the large language models (LLM) using a textual prompt.
Outcome:	The proposed method has been shown to be effective across coarse-grained tasks like image captioning and visual question answering, but more fine-grounded tasks that require spatial understanding have not been thoroughly examined.

Copied to clipboard

Emanuele Bugliarello, Desmond Elliott

Are Scene Graphs Good Enough to Improve Image Captioning? (2020.aacl-main)

Challenge:	Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language.
Approach:	They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization.
Outcome:	The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language.

Copied to clipboard

Victor Milewski, Marie-Francine Moens, Iacer Calixto

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (D19-1)

Challenge:	Existing image captioning models rely on object detection features to generate image descriptions, but they are noisy.
Approach:	They propose to use scene graphs to introduce information about object relations into captioning to improve image descriptions.
Outcome:	The proposed model improves image caption quality by 3.3 CIDEr compared to a strong Bottom-Up Top-Down baseline.

Copied to clipboard

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger

Representation Learning

Challenge:	Existing evaluation metrics are not capable of evaluating text quality.
Approach:	They propose a metric that compares system output against reference texts based on semantics rather than surface forms.
Outcome:	The proposed metric shows a high correlation with human judgment of text quality on a number of text generation tasks.

LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)

Copied to clipboard

Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, Yoon Kim

What Makes for Good Image Captions? (2025.findings-emnlp)

Challenge:	Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view.
Approach:	They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions.
Outcome:	The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations.

Copied to clipboard

Delong Chen, Samuel Cahyawijaya, Etsuko Ishii, Ho Shu Chan, Yejin Bang, Pascale Fung

Improving Reinforcement Learning Based Image Captioning with Natural Language Prior (D18-1)

Challenge:	a formal information-theoretic framework is developed for image captioning . the pyramid of captions is a method that generates enriched captions by integrating local and global visual information.
Approach:	They propose a formal information-theoretic framework for image captioning . they propose 'Pyramid of Captions' method that generates enriched captions .
Outcome:	The proposed framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements.

Copied to clipboard

Tszhang Guo, Shiyu Chang, Mo Yu, Kun Bai

MemeCap: A Dataset for Captioning and Interpreting Memes (2023.emnlp-main)

Challenge:	Recent research shows that Reinforcement Learning (RL) approaches suffer from the exposure bias problem.
Approach:	They propose a Reinforcement Learning (RL) based training framework that constrains the action space using an n-gram language prior.
Outcome:	The proposed model is more human readable and graceful.

Copied to clipboard

EunJeong Hwang, Vered Shwartz

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization (2023.findings-eacl)

Challenge:	a new dataset aims to understand meme captioning tasks using visual metaphors . vision and language models are proving to be effective in image captioning and visual question answering tasks .
Approach:	They present a dataset that contains 6.3K memes and 6.3k meme captions . they show that vision and language models still struggle with visual metaphors despite their advanced capabilities .
Outcome:	The proposed dataset contains 6.3K memes along with the title of the post containing the meme, meme captions, literal image caption, and visual metaphors.

Copied to clipboard

Aishwarya Agrawal, Ivana Kajic, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, Aida Nematzadeh

SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling (2024.eacl-long)

Challenge:	Visual question answering (VQA) is a task of answering open-ended questions about images.
Approach:	They evaluate two vision-and-language (V&L) models under different settings . they find they tend to learn to solve the benchmark rather than the skills required by VQA .
Outcome:	The proposed models exhibit poor generalization under out-of-distribution settings.

Copied to clipboard

Eileen Wang, Caren Han, Josiah Poon

UniRAG: Universal Retrieval Augmentation for Large Vision Language Models (2025.findings-naacl)

Challenge:	Visual storytelling aims to automatically generate a coherent story based on a given image sequence.
Approach:	They propose a framework that represents the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge.
Outcome:	The proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations.

Copied to clipboard

Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, Jimmy Lin

“Wikily” Supervised Neural Translation Tailored to Cross-Lingual Tasks (2021.emnlp-main)

Challenge:	Large Vision Language Models (LVLMs) have unlocked many complex use cases that require Multi-Modal (MM) understanding and MM generation.
Approach:	They propose a plug-and-play technique that adds relevant retrieved information to prompts as few-shot examples during inference.
Outcome:	The proposed method significantly improves the output quality of large vision language models when input prompts are augmented with relevant information retrieved by Vision-Language retrievers like UniRAG.

Copied to clipboard

Mohammad Sadegh Rasooli, Chris Callison-Burch, Derry Tanti Wijaya

Multilinguality

Challenge:	Unsupervised neural machine translation models perform well in low-resource or distant languages.
Approach:	They propose a model that leverages Wikipedia for machine translation and cross-lingual tasks without supervision from external parallel data or supervised models in target language.
Outcome:	The proposed model outperforms supervised models in Arabic and English translation tasks.

What Is Missing in Multilingual Visual Reasoning and How to Fix It (2025.findings-naacl)

Copied to clipboard

Yueqi Song, Simran Khanuja, Graham Neubig

Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (D19-1)

Challenge:	NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users.
Approach:	They propose a translation-test approach to tackle multilinguality, visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality.
Outcome:	The proposed interventions boost open models LLaVA-v1.5-13B by 13.4%, LLva-v1.6-34B by 20.3%, and Qwen-VL by 16.7% while minorly improving GPT-4V’s performance.

Copied to clipboard

Soravit Changpinyo, Bo Pang, Piyush Sharma, Radu Soricut

Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? (2024.emnlp-main)

Challenge:	Object detection is used in vision and language tasks but is expensive to learn . popular models rely on annotating ground-truths for bounding boxes and semantic labels . empirically, object detection leads to effective transfer learning and improved captioning and visual question answering models .
Approach:	They examine the effect of decoupling box proposal and featurization on down-stream tasks . they propose a family of "two-stage" object detectors that propose category-agnostic bounding boxes .
Outcome:	The proposed method improves image captioning and visual question answering models by leveraging large amounts of labeled annotations.

Copied to clipboard

Gregor Geigle, Radu Timofte, Goran Glavaš

Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels (2023.findings-acl)

Challenge:	Large vision-language models (LVLMs) often hallucinate and produce captions that mention concepts that cannot be found in the image.
Approach:	They propose to add grounding objectives to captions that explicitly align image regions or objects to text spans to reduce hallucination.
Outcome:	The proposed evaluation protocol reduces the amount of hallucination in LVLMs by adding grounding objectives.

Copied to clipboard

Bang Yang, Fenglin Liu, Zheng Li, Qingyu Yin, Chenyu You, Bing Yin, Yuexian Zou

Retrieval, Analogy, and Composition: A framework for Compositional Generalization in Image Captioning (2021.findings-emnlp)

Challenge:	Existing approaches to generate informative titles for products with limited labels are inadequate for novel products.
Approach:	They propose a prompt-based approach to generate attractive titles for novel products . they use multimodal prompts to preserve characteristics and writing styles of novel products.
Outcome:	The proposed approach achieves state-of-the-art results on novel product categories with limited labels.

Copied to clipboard

Zhan Shi, Hui Liu, Martin Renqiang Min, Christopher Malon, Li Erran Li, Xiaodan Zhu

Image Caption Generation for News Articles (2020.coling-main)

Challenge:	Existing approaches fail to generalize well to concepts that are not observed during training.
Approach:	They propose a framework that revolves around probing several similar image caption training instances and performing analogical reasoning over relevant entities in retrieved prototypes.
Outcome:	The proposed framework improves on the widely used image captioning benchmarks and on composition-related evaluation metrics.

Copied to clipboard

Zhishen Yang, Naoaki Okazaki

Text-Guided Image Clustering (2024.eacl-long)

Challenge:	Existing work on news-image captioning requires a joint understanding of image and text.
Approach:	They propose a Transformer model that integrates text and image modalities and attends to textual features from visual features in generating a caption.
Outcome:	The proposed model outperforms the state-of-the-art model and improves the quality of news-image captions.

Copied to clipboard

Andreas Stephan, Lukas Miklautz, Kevin Sidak, Jan Philip Wahle, Bela Gipp, Claudia Plant, Benjamin Roth

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022.findings-acl)

Challenge:	Current image clustering methods neglect the use of generated textual descriptions.
Approach:	They propose to use image captioning and visual question-answering to cluster images . they propose a new approach to inject task- or domain knowledge into image clustering .
Outcome:	The proposed method outperforms existing methods on eight image clustering datasets.

Copied to clipboard

Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung

Responsible & Trustworthy NLP

Challenge:	Recent large-scale vision-language pre-training models are powerful in multimodal classification and retrieval tasks.
Approach:	They propose to augment a vision-language pre-training model with a textual pre-trained language model . the model achieves 44.5% zero-shot accuracy on multimodal generation tasks .
Outcome:	The proposed model achieves 44.5% zero-shot accuracy on open-ended visual question answering and image captioning tasks.

On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)

Copied to clipboard

Shruti Palaskar, Akshita Bhagia, Yonatan Bisk, Florian Metze, Alan W Black, Ana Marasovic

Token-level and sequence-level loss smoothing for RNN language models (P18-1)

Challenge:	Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning.
Approach:	They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks.
Outcome:	The proposed model types do not consistently improve self-rationalization in multimodal tasks.

Copied to clipboard

Maha Elbayad, Laurent Besacier, Jakob Verbeek

Object Counts! Bringing Explicit Detections Back into Image Captioning (N18-1)

Challenge:	Maximum likelihood estimation treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space.
Approach:	They propose to extend the reward augmented maximum likelihood approach to token-level loss smoothing by using token-based approaches to improve the model's performance.
Outcome:	The proposed model improves on image captioning and machine translation tasks and treats all sentences that do not match the ground truth as poor .

Copied to clipboard

Josiah Wang, Pranava Swaroop Madhyastha, Lucia Specia

DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training (2022.findings-acl)

Challenge:	Existing approaches to image captioning use explicit object detectors as an intermediate step, but they bypass the explicit detection phase and instead generate captions directly from image embeddings.
Approach:	They argue that explicit detections provide rich semantic information and can thus be used as an interpretable representation to better understand why end-to-end image captioning systems work well.
Outcome:	The proposed methods can be used to understand why end-to-end captioning systems work well.

Copied to clipboard

Luyang Huang, Guocheng Niu, Jiachen Liu, Xinyan Xiao, Hua Wu

TROPE: TRaining-Free Object-Part Enhancement for Seamlessly Improving Fine-Grained Zero-Shot Image Captioning (2024.findings-emnlp)

Challenge:	Existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation due to the limitations of the model structure and pre-training objectives.
Approach:	They propose a framework which unifies vision-and-language generation as sequence generation problems.
Outcome:	The proposed framework achieves better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss on image captioning and text-to-image generation datasets.

Copied to clipboard

Joshua Feinglass, Yezhou Yang

Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models (2025.coling-main)

Challenge:	Existing approaches to enhance zero-shot abilities in image captioning fail with fine-grained datasets.
Approach:	They propose a method to enhance captions with additional object-part details using object detector proposals and natural language processing techniques.
Outcome:	The proposed method improves performance on fine-grained datasets and improves on existing methods.

Copied to clipboard

Zijun Chen, Wenbo Hu, Guande He, Zhijie Deng, ZHeng ZHang, Richang Hong

Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)

Challenge:	Multimodal large language models combine visual and textual data for tasks like image captioning and visual question answering.
Approach:	They propose temperature scaling and iterative prompt optimization to calibrate MLLMs and enhance model reliability.
Outcome:	The proposed techniques improve MLLMs and improve model reliability.

Copied to clipboard

Ozan Caglayan, Pranava Madhyastha, Lucia Specia

On Hallucination and Predictive Uncertainty in Conditional Language Generation (2021.eacl-main)

Challenge:	a few popular metrics are still used to evaluate language generation systems despite their known limitations.
Approach:	They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts .
Outcome:	The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set.

Copied to clipboard

Yijun Xiao, William Yang Wang

G-Cap: A Game Character Caption Generator (2026.acl-long)

Challenge:	Modern deep neural network models have brought drastic improvements in generation quality measured by standard metrics on different natural language generation tasks.
Approach:	They propose a beam search extension to reduce hallucination in conditional language generation by adding a prediction extension to beam search.
Outcome:	The proposed extension improves trading performance on standard metric for less hallucination with the proposed beam search variant.

Copied to clipboard

Yang Yang, Feng Hu, Haiming Zhang, XU Cheng, Gui Zheng, Liang Yao, Wenqi Ren

Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2021.eacl-main)

Challenge:	Existing studies on Large Vision-Language Models (LVLMs) primarily focus on real-world scenarios, leaving surreal, highly stylized, and semantically hybrid virtual-world situations significantly underexplored.
Approach:	They propose to use a manually annotated benchmark to evaluate LVLMs' ability to perceive and describe game character from the virtual-world.
Outcome:	The proposed task evaluates LVLMs’ ability to perceive and describe game character from the virtual-world.

Copied to clipboard

Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, Yinfei Yang

Transparent Human Evaluation for Image Captioning (2022.naacl-main)

Challenge:	Existing image captioning datasets have limited cross-modal associations, preventing researchers from examining how inter-modal learning impacts intra-modal tasks.
Approach:	They propose to use image captioning data to support multi-modal retrieval training and evaluation to assess the impact of inter-modality learning.
Outcome:	The proposed model is able to measure the influence of intra- and inter-modality learning.

Copied to clipboard

Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, Noah A. Smith

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)

Challenge:	Recent work has demonstrated that image captioning is a complex task that requires a large amount of human input.
Approach:	They develop a human evaluation protocol for image captioning models based on machine- and human-generated captions on the MSCOCO dataset.
Outcome:	The proposed model improves CLIPScore, a recent metric that uses image features, and improves human judgments because it is more sensitive to recall.

Copied to clipboard

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith

Retrieval-augmented Image Captioning (2023.eacl-main)

Challenge:	Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models .
Approach:	They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation.
Outcome:	The proposed leaderboards track progress in language generation models and metrics for their evaluation.

Copied to clipboard

Rita Ramos, Desmond Elliott, Bruno Martins

Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts (P19-1)

Challenge:	Experimental results show that image captioning can be effectively formulated from this new perspective.
Approach:	They propose a pretrained visual and language decoders for image captioning that generate sentences from the input image and a set of captions retrieved from a datastore.
Outcome:	The proposed model generates sentences given the input image and retrieved captions, while the decoder attends to the multimodal encoder representations.

Copied to clipboard

Alakananda Vempala, Daniel Preoţiuc-Pietro

Chart-to-Text: A Large-Scale Benchmark for Chart Summarization (2022.acl-long)

Challenge:	Social media posts often contain images to provide content, provide context, or express feelings.
Approach:	They build and release a dataset of image tweets annotated with four different classes which express whether the text or the image provides additional information to the other modality.
Outcome:	The proposed method can be used in several downstream applications including pre-training image tagging models and collecting distantly supervised data for image captioning.

Copied to clipboard

Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty

COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)

Challenge:	Inferring key insights from charts can be challenging and time-consuming.
Approach:	They propose a task where the goal is to explain a chart and summarize key takeaways from it in natural language.
Outcome:	The proposed model produces fluent summaries but suffers from hallucinations and factual errors . the proposed model is compared with other models and can be used to generate BLEU scores .

Copied to clipboard

Mert Inan, Piyush Sharma, Baber Khalid, Radu Soricut, Matthew Stone, Malihe Alikhani

iParaphrasing: Extracting Visually Grounded Paraphrases via an Image (C18-1)

Challenge:	Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric.
Approach:	They propose a first discourse-aware learned generation metric for evaluating image descriptions.
Outcome:	The proposed metric predicts human ratings of captions on out-of-domain images.

Copied to clipboard

Chenhui Chu, Mayu Otani, Yuta Nakashima

AGIC: Attention-Guided Image Captioning to Improve Caption Relevance (2026.findings-eacl)

Challenge:	iParaphrasing extracts visually grounded paraphrases, which are different phrasal expressions describing the same visual concept in an image.
Approach:	They propose a task to extract visually grounded paraphrases from images . they propose to model the similarity between the extracted VGPs using existing methods .
Outcome:	The proposed task extracts visually grounded paraphrases from images . the proposed method has the potential to improve multimodal language and image tasks .

Copied to clipboard

L. D. M. S. Sai Teja, Ashok Urlana, Pruthwik Mishra

Generating Question Relevant Captions to Aid Visual Question Answering (P19-1)

Challenge:	Existing methods for image captioning generate generic captions that are limited in capturing nuanced visual details.
Approach:	They propose attention-guided image captioning which amplifies visual regions directly in the feature space to guide caption generation.
Outcome:	The proposed approach matches or surpasses state-of-the-art models while achieving faster inference.

Copied to clipboard

Jialin Wu, Zeyuan Hu, Raymond Mooney

Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation (D18-1)

Challenge:	Visual question answering and image captioning require a shared body of general knowledge connecting language and vision.
Approach:	They propose a method that exploits a shared body of general knowledge connecting language and vision by jointly generating captions.
Outcome:	The proposed approach obtains state-of-the-art performance on the VQA v2 challenge . it uses human annotated captions to generate question-relevant captions .

Copied to clipboard

Renjie Zheng, Mingbo Ma, Liang Huang

LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024.naacl-long)

Challenge:	Neural text generation has been quite successful recently, but during training time, only one reference is considered for each example, even though there are often multiple references available.
Approach:	They propose an algorithm to generate exponentially many pseudo-references by compressing existing references into lattices and traversing them to generate new pseudo-References.
Outcome:	The proposed model significantly improves on baselines in machine translation and image captioning, and is comparable to existing models.

Copied to clipboard

Yuchi Wang, Shuhuai Ren, Rundong Gao, Linli Yao, Qingyan Guo, Kaikai An, Jianhong Bai, Xu Sun

CapEEN: Image Captioning with Early Exits and Knowledge Distillation (2024.findings-emnlp)

Challenge:	Existing models for text-to-image generation have been underperforming in image-totext generation tasks.
Approach:	They propose a framework that uses a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths.
Outcome:	The proposed framework achieves state-of-the-art performance on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr .

Copied to clipboard

Divya Jyoti Bajpai, Manjesh Hanawal

SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data (2025.acl-long)

Challenge:	Early Exit (EE) strategies can be used to enhance their efficiency, but their adaptation presents challenges in image captioning as it requires varying levels of semantic information for accurate predictions.
Approach:	They propose a framework to improve the performance of EE strategies by knowledge distillation . they use a variant A-CapEEN to adapt thresholds on the fly to account for drifts .
Outcome:	The proposed framework gains speedup of 1.77 while maintaining competitive performance compared to the final layer.

Copied to clipboard

Michael Ogezi, Freda Shi

QACE: Asking Questions to Evaluate an Image Caption (2021.findings-emnlp)

Challenge:	Vision-language models struggle with spatial reasoning, a skill that humans excel at.
Approach:	They propose to use a spatial-reasoning Enhanced (SpaRE) VLM to improve spatial reasoning in visual question answering and robotics.
Outcome:	The proposed model achieves a 49% performance gain on the What's Up benchmark while maintaining strong results on general tasks.

Copied to clipboard

Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, Kyomin Jung

Probing Cross-modal Semantics Alignment Capability from the Textual Perspective (2022.findings-emnlp)

Challenge:	Existing metric for image captioning evaluation is based on n-gram similarity metrics but these fail to capture semantic errors in captions.
Approach:	They propose a new metric based on Question Answering for Caption Evaluation to evaluate image captioning based upon Question Generation and Question Answers systems.
Outcome:	The proposed metric is multi-modal, reference-less and explainable.

Copied to clipboard

Zheng Ma, Shi Zong, Mianzhi Pan, Jianbing Zhang, Shujian Huang, Xinyu Dai, Jiajun Chen

Divide and Conquer Radiology Report Generation via Observation Level Fine-grained Pretraining and Prompt Tuning (2024.emnlp-main)

Challenge:	In recent years, vision and language pre-training (VLP) models have advanced the state-of-the-art results in a variety of cross-modal downstream tasks.
Approach:	They propose a new probing method that is based on image captioning to first empirically study the cross-modal semantics alignment of VLP models.
Outcome:	The proposed method analyzes captions generated by five popular VLP models to reveal how well they align with visual words and how well these align with images.

Copied to clipboard

Yuanpin Zhou, Huogen Wang

VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions (2025.emnlp-main)

Challenge:	Recent advances in image captioning and vision-language pretraining have made it difficult for radiologists to generate coherent and accurate reports.
Approach:	They propose a model which breaks down full-text radiology reports into concise observation descriptions and encodes observation predictions into a decoding stage.
Outcome:	The proposed model achieves significant improvements across all metrics, underscoring its capability to generate semantically coherent and clinically accurate radiology reports.

Copied to clipboard

Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Seitaro Otsuki, Komei Sugiura

On Vision Features in Multimodal Machine Translation (2022.acl-long)

Challenge:	Existing evaluation metrics for image captioning are primarily designed for short captions and are not suitable for long captions.
Approach:	They propose an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework.
Outcome:	The proposed metric outperforms existing metrics and achieves superhuman performance on LongCap-Arena.

Copied to clipboard

Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, JingBo Zhu

Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization (2024.acl-long)

Challenge:	Recent work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is given to the quality of vision models.
Approach:	They develop a selective attention model to study the patch-level contribution of an image in multimodal machine translation.
Outcome:	The proposed model is able to learn translation from the visual modality on probing tasks and is compared with existing models.

Copied to clipboard

Takumi Takada, Yuma Suzuki, Hiroki Takushima, Hayato Tanoue, Haruki Sato, Aiswariya Kumar, Hiroki Nishihara, Takayuki Hori, Kazuya Ueki

Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributes (2024.emnlp-main)

Challenge:	Recent large-scale vision language models (VLMs) lack continuity between learning objective and performance metrics.
Approach:	They propose a lightweight final-metric-optimizing training method that replaces the expensive exploration process in RL with an offline, diverse text data augmentation method.
Outcome:	The proposed method achieves comparable performance to state-of-the-art RL method while saving hundreds of times more model forwarding iterations and greater amounts of computation time.

Copied to clipboard

Yusuke Hirota, Jerone Andrews, Dora Zhao, Orestis Papakyriakopoulos, Apostolos Modas, Yuta Nakashima, Alice Xiang

Information Retrieval

Challenge:	Traditional approaches only target labeled attributes, ignoring biases from unlabeled ones.
Approach:	They propose a method that ensures protected group independence from all attributes and mitigates inpainting biases through data filtering.
Outcome:	The proposed approach achieves an average reduction of 46.1% in leakage-based bias metrics for multi-label classification and 74.8% for image captioning.

LIME: Less Is More for MLLM Evaluation (2025.findings-acl)

Copied to clipboard

King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shuyue Guo, Tianyu Zheng, Jiawei Guo, Bo Li, Haoning Wu, Xingwei Qu, Jian Yang, Ruibo Liu, Xiang Yue, Jiaheng Liu, Chenghua Lin, Hamid Alinejad-Rokny, Min Yang, Shiwen Ni, Wenhao Huang, Ge Zhang

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022.emnlp-main)

Challenge:	Existing MLLM benchmarks and unified evaluation frameworks cannot accurately and efficiently reflect the ability of MLMLs.
Approach:	They propose a semi-automated benchmark curated using a pipeline that filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding.
Outcome:	The proposed benchmark reduces the number of samples by 76% and evaluation time by 77% while it can more effectively distinguish different models’ abilities.

Copied to clipboard

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)

Challenge:	Existing pre-trained vision-language models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment.
Approach:	They propose a vision-language foundation model with cross-modal skip-connections that can be pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives.
Outcome:	The proposed model achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering.

Copied to clipboard

Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, William Yang Wang

BERTGen: Multi-task Generation through BERT (2021.acl-long)

Challenge:	Existing learning metrics are limited to tasks where large human ratings are available.
Approach:	They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation.
Outcome:	The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation.

Copied to clipboard

Faidon Mitzalis, Ozan Caglayan, Pranava Madhyastha, Lucia Specia

Understanding Retrieval Robustness for Retrieval-augmented Image Captioning (2024.acl-long)

Challenge:	Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU).
Approach:	They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks.
Outcome:	The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts.

Copied to clipboard

Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, Desmond Elliott

Hongkuan Zhang, Saku Sugawara, Akiko Aizawa, Lei Zhou, Ryohei Sasano, Koichi Takeda

Challenge:	Recent retrieval-augmented models for image captioning are not perfect in practice.
Approach:	They propose to train a retrieval-augmented captioning model SmallCap by sampling retrieved captions from more diverse sets.
Outcome:	The proposed model is sensitive to tokens that appear in the majority of retrieved captions . the proposed model improves both in-domain and cross-domain performance .

Copied to clipboard

JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation (2022.coling-1)

Challenge:	Existing image captioning approaches treat image-caption pairs indistinctly without considering the differences in their learning difficulties.
Approach:	They propose a pretrained vision–language model that measures cross-modal similarity and a model that uses cross-module similarity to measure the difficulty of captioning.
Outcome:	The proposed model achieves superior performance and competitive convergence speed to baselines without incurring additional training costs.

Copied to clipboard

Jingyi You, Dongyuan Li, Manabu Okumura, Kenji Suzuki

Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)

Challenge:	Existing methods rarely consider cross-modal alignment between textual and visual features and ignore disease tags as auxiliary for report generation.
Approach:	They propose a "Jointly learning framework for automated disease Prediction and radiology report Generation" the framework integrates cross-modal alignment between textual and visual features and disease tags to improve the quality of reports.
Outcome:	The proposed framework improves the quality of radiology reports by combining the main task and auxiliary tasks.

Copied to clipboard

Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, Matthew Stone

SilVar: Speech-Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization (2025.emnlp-main)

Challenge:	Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models .
Approach:	They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning.
Outcome:	The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show .

Copied to clipboard

Tan-Hanh Pham, Le Hoang Nam, Phu-Vinh Nguyen, Chris Ngo, Truong-Son Hy

Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks (2022.emnlp-main)

Challenge:	Visual Language Models have demonstrated remarkable capabilities across various tasks, including visual question answering and image captioning.
Approach:	They propose an end-to-end multimodal model that leverages speech instructions for reasoning-based visual question answering.
Outcome:	The proposed model can process and explain visual scenes from spoken input, moving beyond simple object recognition to reasoning-based interactions.

Copied to clipboard

Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, Daniel Whitenack

IRR: Image Review Ranking Framework for Evaluating Vision-Language Models (2025.coling-main)

Challenge:	In total, the initial release of the Bloom Library datasets covers 363 languages across 32 language families.
Approach:	They present a set of multimodal and multilingual datasets for language modeling, image captioning, visual storytelling, and speech synthesis/recognition.
Outcome:	The Bloom Library datasets cover 363 languages across 32 language families.

Copied to clipboard

Kazuki Hayashi, Kazuma Onishi, Toma Suzuki, Yusuke Ide, Seiji Gobara, Shigeki Saito, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020.emnlp-main)

Challenge:	Large-scale vision language models excel at generating factual content, but their ability to rank images from multiple perspectives has not been explored.
Approach:	They propose a framework to evaluate large-scale vision-language models by measuring their ability to rank image texts from multiple perspectives.
Outcome:	The proposed evaluation framework measures how closely LVLMs' judgments align with human interpretations.

Copied to clipboard

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi

FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning (2022.emnlp-main)

Challenge:	Recent work has adapted vision-and-language models to generative tasks like image captioning.
Approach:	They propose an extension to LXMERT with training refinements to generate images from text.
Outcome:	The proposed model can generate images from pieces of text while still being comparable to existing models.

Copied to clipboard

Suvir Mirchandani, Licheng Yu, Mengjiao Wang, Animesh Sinha, Wenwen Jiang, Tao Xiang, Ning Zhang

CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)

Challenge:	Prior work on multimodal fashion tasks has been limited by the data in individual benchmarks or has leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fashion data.
Approach:	They propose a fashion-specific pre-training framework based on weakly-supervised triplets constructed from fashion image-text pairs.
Outcome:	The proposed framework is based on weakly-supervised triplets constructed from fashion image-text pairs and is competitive on a diverse set of fashion tasks.

Copied to clipboard

Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen

IDC: Boost Text-to-image Retrieval via Indirect and Direct Connections (2024.lrec-main)

Challenge:	Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected.
Approach:	They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans .
Outcome:	The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test .

Copied to clipboard

Guowei Ge, Kuangrong Hao, Lingguang Hao

Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021.emnlp-main)

Challenge:	Dual Encoders (DE) and Cross Attention (CA) frameworks for image and text retrieval are more accurate but slower.
Approach:	They propose a dual encoders-based approach to map image and text inputs into a coordinated representation space and calculate their similarity directly.
Outcome:	Extensive experiments on the MSCOCO and Flickr30K datasets validate the effectiveness of the proposed methods.

Copied to clipboard

Pierre Colombo, Guillaume Staerman, Chloé Clavel, Pablo Piantanida

PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning (2023.findings-emnlp)

Challenge:	metric BaryScore is used to evaluate text generation based on deep contextualized embeddings.
Approach:	They propose to model the layer output of deep contextualized embeddings as a probability distribution rather than a vector embeddable layer.
Outcome:	The proposed metric outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

Copied to clipboard

Yongil Kim, Yerin Hwang, Hyeongu Yun, Seunghyun Yoon, Trung Bui, Kyomin Jung

ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding (2023.findings-emnlp)

Challenge:	Existing image captioning metrics are vulnerable to lexical perturbations, but they are not robust to such perturbations.
Approach:	They propose a perturbation-robust multilingual CLIPScore which is a reference-free image captioning metric for multiple languages.
Outcome:	The proposed metric outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages while maintaining a strong correlation with human judgments.

Copied to clipboard

Guojun Wu

Treble Counterfactual VLMs: A Causal Approach to Hallucination (2025.findings-emnlp)

Challenge:	Existing models that use multilingual captions for images have limited results due to the scarcity of training data.
Approach:	They propose a multilingual vision-and-language model that divides a V&L task into two stages . they propose IC, which takes the caption as the alt text and performs cross-lingual language understanding .
Outcome:	The proposed model can achieve state-of-the-art results for five languages and comparable results for the rest.

Copied to clipboard

Li Li, Jiashu Qu, Linxin Song, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao

Puzzled by Puzzles: When Vision-Language Models Can’t Take a Hint (2025.emnlp-main)

Challenge:	Existing studies link hallucination to data or representation biases, but their causal origins remain unclear.
Approach:	They propose a causal framework to analyze and mitigate hallucination in vision-language models by using counterfactual analysis to estimate the Natural Direct Effect (NDE) of each modality and their interaction.
Outcome:	The proposed framework significantly reduces hallucination while preserving task performance while retaining reliability.

Copied to clipboard

Heekyung Lee, Jiaxin Ge, Tsung-Han Wu, Minwoo Kang, Trevor Darrell, David M. Chan

Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning (2024.lrec-main)

Challenge:	rebus puzzles encode language through imagery, spatial arrangement, and symbolic substitution.
Approach:	They construct a benchmark of rebus puzzles in english language to test their ability to interpret and solve them.
Outcome:	The proposed model performs well on a set of english-language rebus puzzles.

Copied to clipboard

Guisheng Liu, Yi Li, Zhengcong Fei, Haiyan Fu, Xiangyang Luo, Yanqing Guo

IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning (2024.emnlp-main)

Challenge:	Existing image captioning models require large trainable parameters to bridge visual and textual representations.
Approach:	They propose a lightweight image captioning network in combination with continuous diffusion that injects prefix image embeddings into denoising process of diffusion model.
Outcome:	The proposed method generates diverse captions with relatively less parameters while maintaining fluency and relevance compared with other models.

Copied to clipboard

Soeun Lee, Si-Woo Kim, Taewhan Kim, Dong-Jin Kim

Releasing the Capacity of GANs in Non-Autoregressive Image Captioning (2024.lrec-main)

Challenge:	Existing text-only training methods overlook the modality gap between using text data during training and employing images during inference.
Approach:	They propose a novel approach that aligns text features with visually relevant features to mitigate the modality gap between using text data during training and employing images during inference.
Outcome:	The proposed method outperforms the state-of-the-art methods in image captioning and video captioning by a significant margin compared to training with text data.

Copied to clipboard

Da Ren, Qing Li

Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)

Challenge:	Existing non-autoregressive (NAR) models suffer from their inherent multi-modality problem.
Approach:	They propose an Adversarial Non-autoregressive Transformer for Image Captioning that improves model performance by modifying model structure to be compatible with contrastive learning.
Outcome:	The proposed model achieves 26.72 times faster than the autoregressive model on the MSCOCO dataset.

Copied to clipboard

Mong Yuan Sim, Wei Emma Zhang, Xiang Dai, Biaoyan Fang

Nearest Neighbor Normalization Improves Multimodal Retrieval (2024.emnlp-main)

Challenge:	Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions.
Approach:	They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding.
Outcome:	The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information.

Copied to clipboard

Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, Tristan Thrush

Mitigating Open-Vocabulary Caption Hallucinations (2024.emnlp-main)

Challenge:	Recent training-free methods suggest that accuracy can be improved without fine-tuning.
Approach:	They propose a method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization.
Outcome:	The proposed method improves retrieval metrics for all contrastive models and datasets and does not require training on the reference database.

Copied to clipboard

Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, Hadar Averbuch-Elor

Reasoning

Challenge:	Existing methods for image captioning ignore the long-tailed nature of hallucinations . a new framework is proposed to address hallucines in image captions in the open-vocabulary setting .
Approach:	They propose a framework to address hallucinations in image captioning in the open-vocabulary setting.
Outcome:	The proposed framework surpasses the CHAIR benchmark in diversity and accuracy in open-vocabulary captioning.

EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)

Copied to clipboard

Hyunjong Kim, Sangyeop Kim, Jongheon Jeong, Yeongjae Cho, Sungzoon Cho

Explainability & Interpretability in NLP

Challenge:	Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified.
Approach:	They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness.
Outcome:	The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics.

Text360Nav: 360-Degree Image Captioning Dataset for Urban Pedestrians Navigation (2024.lrec-main)

Copied to clipboard

Chieko Nishimura, Shuhei Kurita, Yohei Seki

Inject to Heal: Alleviating hallucination in LVLMs via Context Embedding Injection (2026.findings-acl)

Challenge:	Existing image captioning datasets focus on the overall image description and lack detailed scene descriptions, overlooking features for pedestrians walking on urban streets.
Approach:	They develop a dataset to provide textual feedback from 360-degree camera images to visually impaired pedestrians . they generate meaningful captions focusing on obstacles on the streets .
Outcome:	The proposed dataset provides textual feedback from machinery visual perception to visually impaired individuals and distracted pedestrians . the results show that the models trained with the dataset can generate meaningful captions focusing on street objects and obstacles in urban scenes .

Copied to clipboard

Mehrdad Fazli, Bowen Wei, Ziwei Zhu

CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation (2026.acl-long)

Challenge:	a large vision-language model can generate hallucinations inconsistent with visual input . a lightweight method that embeds the last input token as a grounding signal reduces the likelihood of hallucinosity.
Approach:	They propose a training-free mitigation strategy that harnesses the hidden state of the last input token as a grounding signal to maintain visual fidelity throughout decoding and curb hallucinations.
Outcome:	The proposed method outperforms state-of-the-art methods on CHAIR, AMBER, and MMHal benchmarks.

Copied to clipboard

Ashish Mishra, Tarun Kumar, Arpit Shah, Suparna Bhattacharya, Martin Foltin